Evaluation Across Contexts: Evaluating the Impact of Technology Integration 
Professional Development Partnerships 


Louanne Smolin 

National Louis University 


Kimberly A. Lawless 

University of Illinois at Chicago 


Abstract 

Professional development is a neces¬ 
sary component for effectively inte¬ 
grating technology into classrooms. 
Unfortunately, the evaluation of tech¬ 
nology integration professional devel¬ 
opment (TIPD) rarely moves beyond 
participation satisfaction surveys, 
nor does it reflect the concerns of the 
multiple stakeholders participating in 
technology integration efforts. In this 
article, the authors discuss collab¬ 
orative models that hold potential for 
evaluating TIPD partnerships. They 
advocate for TIPD partners to define 
a collaborative and holistic vision 
of success that can guide the evalu¬ 
ation process. The authors discuss 
three specific collaborative evaluation 
models, examine key issues associ¬ 
ated with implementing them, and 
analyze how each model has the po¬ 
tential to strengthen and sustain pro¬ 
fessional development partnerships. 
(Keywords: Technology integration, 
professional development, program 
evaluation) 


I n his informative work on program 
evaluation, Patton maintains “that so¬ 
cial science has proven especially inept 
at offering solutions for the great prob¬ 
lems of our time.... There is a pressing 
need to make headway with these large 
challenges and push the boundaries of 
social innovation to make real progress” 
(Patton, 2006, p. 28). One arena where 
there is great potential for pushing the 
boundaries of social innovation forward 
is integrating technology in schools. 
Students are already well versed and facile 
with using technology to shape their 
worlds outside of school (Jones, Johnson- 
Yale, Perez, & Schuler, 2007). The poten¬ 
tial for technology to impact students in 
school should be realized as well. Teach¬ 


ers can use technology to transform the 
teaching and learning context in a way 
that will position their students for future 
opportunities in the global context, pre¬ 
paring them for the flattened world that 
technology has helped to make possible. 
Through technology, teachers and stu¬ 
dents can soften the boundaries between 
life in schools and in communities as well 
as between their present and future lives. 
Technology has the potential to expand 
learning in ways that a traditional cur¬ 
riculum cannot. 

Yet the evaluation of technology 
integration, including professional 
development for technology integration, 
has done little to define what constitutes 
effective practices for realizing such 
potential. So although the images of our 
classrooms have significantly changed 
due to the ubiquity of technology, and 
many teachers are incorporating tech¬ 
nology in their learning environments, 
these changes have done little to truly 
reform education. In essence, the “more 
things change, the more they remain the 
same” (Sarason, 1996, p. 338). 

This article explores the possibilities 
for collaborative evaluation of technol¬ 
ogy integration professional development 
(TIPD) to transform technology practices 
in schools. To achieve this transforma¬ 
tion, we need to shift and expand the re¬ 
search and evaluation processes we use to 
learn about how TIPD impacts classroom 
practices and student learning. 

Expanding Current Evaluation Models 
of Technology Integration Professional 
Development (TIPD) 

In their provocative review of research, 
“Professional Development in Inte¬ 
grating Technology into Teaching and 
Learning,” Lawless and Pellegrino (2007) 
advocate that the evaluation of TIPD 


can facilitate changes to teaching and 
learning in schools. They maintain that 
although there is a strong perceived 
need for action in terms of TIPD, 
the knowledge base derived through 
research does not guide it. In particular, 
they advocate for more careful and more 
systematic approaches for document¬ 
ing how technology integration oc¬ 
curs within schools, what increases its 
adoption by teachers, and the long-term 
impacts that these investments have on 
teachers and students. Based on this, 
they propose a sequential three-phase 
evaluation design that includes evalua¬ 
tion of (1) program characteristics, (2) 
teacher outcomes, and (3) sustained 
teacher change and student achievement 
effects. They maintain that these phases 
should be sequential. They contend 
that more needs to be known about the 
varieties of program structures, such as 
mentoring or train the trainers’ models, 
before student and teacher outcomes can 
be evaluated. With this sequence, causal 
links between specific program variables 
and outcomes can be substantiated. 
Within each of these phases, they raise 
key questions that should underscore 
the design of evaluation and possible 
outcomes of such an evaluation. 

Desimone (2009) proposes a 
similar, outcomes-based professional 
development evaluation model. She 
states that the goal when evaluat¬ 
ing professional development is to 
move beyond participant satisfaction 
surveys and to systematically measure 
its impact on teachers and students 
in a coherent way. However, rather 
than a sequential model, she advocates 
for a connected model in which the 
evaluation of program characteristics, 
teacher practice, and student outcomes 
are recursive. Such analysis can help 
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evaluators examine relationships among 
what is taught in professional develop¬ 
ment, what occurs in classrooms, and 
how it affects students. 

Both models underscore the necessity 
of analyzing the impact of professional 
development program activities on both 
teacher learning and student achieve¬ 
ment. They are noteworthy in their 
attempts to create connected evalua¬ 
tions that link program inputs to teacher 
and student outcomes. Yet they are 
incomplete. Both models leave out the 
importance of the number of stakehold¬ 
ers involved in technology integration, 
as well as the impact of the stakehold¬ 
ers’ collaborative relationships on the 
meaningful integration of technology 
in today’s schools. They focus on what 
is to be evaluated rather than how. We 
must build on these models, incorporat¬ 
ing aspects of partnership, to deepen 
and sustain the impact of TIPD. In our 
evaluations, we must take into account 
how to foster long-term partnerships 
where all partners benefit. 

A myriad of stakeholders is involved 
in the integration of technology in 
schools, including funders, teachers, 
school administrators, ICT personnel, 
higher education institutions, parents, 
and community members. Working 
together, these collaborators can garner 
sustained funding. They can engage in 
joint planning that benefits a continuum 
of learners from preservice teachers 
and inservice teachers to K-12 stu¬ 
dents. Through collaborative exchanges, 
university-based content area experts 
and classroom-based teacher practitio¬ 
ners can maintain a focus on integrated 
academic content rather than decon- 
textualized technology skill building. 
Therefore, in a professional development 
partnership, a balance of learning occurs. 
As such, teachers have the benefit of 
working closely with content area experts, 
and university content area experts have 
the benefit of learning about pedagogical 
practice situated in local contexts. In this 
way, professional development approach¬ 
es can be tailored and individualized to 
participants’ needs. Teachers have greater 
potential to apply what they are learning 
in professional development programs to 


their specific classrooms, and teacher ed¬ 
ucators can tailor their methods courses 
to better prepare teacher candidates for 
classroom-based pedagogies. 

Yet often the evaluation of TIPD is 
not a collaborative endeavor. Of the 
stakeholders described above, we have 
been concerned with three primary 
groups: funders, institutions of higher 
education, and teachers. From their 
vantage points, each of these stakehold¬ 
ers has defined their own vision for what 
constitutes effective TIPD, which lacks a 
holistic view. 

For example, funders often deem 
TIPD successful if it affects student learn¬ 
ing. Technology integration is a costly, 
ongoing expense. External sources of 
funding are necessary to maintain not 
only meaningful technology integration, 
but also functional technology-infused 
learning environments. Funders have 
a stake in knowing whether and how 
their support affects teacher and student 
learning. Funders want to know whether 
the benefits of their technology fund¬ 
ing outweigh the costs (Kleiman, 2004; 
Lemke & Coughlin, 1998). Yet often 
the outcomes of collaborative teacher 
professional development are difficult 
to measure because they do not tend to 
provide tangible and immediate evidence 
(Lieberman & Grolnick, 1996). Therefore, 
they might not reflect what funders are 
interested in knowing. Specifically, a 
funder’s vision of successful TIPD is one 
that quantifies the difference that their 
funds have made with respect to student 
learning, and without tangible evidence 
this could be difficult to establish. 

Institutions of higher education 
play two important roles: They have 
the potential to create new frontiers for 
transforming teaching and learning en¬ 
vironments with the use of technology, 
and they provide a pipeline of technol¬ 
ogy-proficient inservice and preservice 
teachers who can implement these 
transformations and become leader 
advocates for technology integration 
across university and school contexts 
(Strudler & Weitzel, 1999). Their vi¬ 
sion of successful TIPD is the extent to 
which participants use the transforma¬ 
tive technology-infused practices they 


teach, the quality of the participants’ 
technology-integration practices, and 
the resulting student outcomes. 

Teachers are “frontline” stakehold¬ 
ers. Through their roles, they can adjust 
and fine-tune what they have learned 
in professional development sessions to 
their unique contexts and students. For 
them, success is inextricably bound to 
their teaching and classroom culture. 
Teachers deem professional development 
successful if it deepens their teaching of 
a particular concept, helps them create 
instructional conditions conducive to 
student engagement, and fostering stu¬ 
dent learning of content (Mumtaz, 2000). 
So it is through their direct work with 
students that they can incorporate what 
they have learned within their teaching 
practice and implement transforma¬ 
tive technology practices within their 
classrooms. Ironically, teachers’ roles in 
evaluation are typically as limited, pas¬ 
sive respondents. 

These individualistically derived 
notions of success have resulted in 
short-term changes and have hampered 
long-term improvements in teaching 
and learning. Teachers have difficulty 
sustaining the transformative practices 
they learn in professional develop¬ 
ment without ongoing support and 
mentorship. As such, their potential for 
affecting their students’ learning, as well 
as their mentorship of new teachers, is 
difficult to achieve. Higher education 
partners lose an important laboratory 
of innovation as well as placements for 
their students. When success cannot 
be sustained long-term, funders are 
hesitant to continue their support. As a 
result, teaching and learning may revert 
back to the status quo. 

Lack of collaboration also impacts 
the evaluation of TIPD. The traditional 
program evaluation scenario is one in 
which an external “expert” evaluator 
carefully collects information about a 
program or some aspect of a program 
to make necessary decisions about the 
program. This approach does not high¬ 
light the relationships evaluators must 
develop to collect data and information, 
nor does it detail who should be mak¬ 
ing decisions related to the evaluation, 
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Table 1 . Collaborative Evaluation Models 


Evaluation Model 

Focus of Evaluation 

Key Processes 

Developmental 

Evaluation 

Continuous program growth and progress 

Dialogic, facilitating emergent evaluation 
designs that nurture program growth 

Responsive Evaluation 

Providing assistance to educators/ 
participants that is situated in their class¬ 
room contexts 

Collaborative, gaining the multiple perspec¬ 
tives of all stakeholders concerning data 
collection strategies and interpretations of 
findings 

Layered Research 

Expanding the knowledge base concerning 
effective classroom practice based on 
participant’s inquiries and contexts 

Action research/teacher inquiry, including 
participant-driven documentation 


who should have access to the results, 
and how they should implement rec¬ 
ommendations. Absent these consider¬ 
ations, a danger exists that evaluation 
outcomes will be irrelevant or impracti¬ 
cal for all stakeholders. 

We advocate collaborative and holis¬ 
tic visions of success rather than sepa¬ 
rate visions characteristic of traditional 
approaches to evaluation. Through 
collaboration stakeholders can arrive 
at a shared vision of success through 
evaluation. In so doing, stakeholders can 
change and enlarge the lenses through 
which they evaluate success. 

Collaborative evaluation involves all 
stakeholders asking common ques¬ 
tions such as “What is it that teachers 
need to have to successfully integrate 
technology?” “What do higher educa¬ 
tion partners need for success?” and 
“What demonstrative proof do funders 
need?” Stakeholders must also explore 
what evaluation processes foster col¬ 
laboration, how information should be 
gathered, and how it should be shared. 
Evaluations grounded in shared ques¬ 
tioning ensure that all stakeholders’ 
needs are met and that the definitions 
of success are mutually valued and col- 
laboratively derived. 

When evaluation becomes a collab¬ 
orative endeavor, boundaries between 
the various stakeholders can be crossed. 
For example, technology integration is 
a field-based process, and the contex¬ 
tual realities surrounding technology- 
integration efforts may be unique from 
participant to participant. Evaluation of 
TIPD can uncover the contextual reali¬ 
ties surrounding a program as it is being 
implemented in the field. Stakeholders 
can then arrive at a shared understand¬ 
ing of what is going on in the field and 


draw upon the field when designing 
instruments, analyzing results, and of¬ 
fering recommendations that are feasible 
to integrate in practice. 

Collaborative Approaches that Can 
Inform TIPD Evaluation 

How can collaborative evaluation be 
achieved when it is “more often messy, 
not orderly, emergent, not controlled” 
(Patten, 2006, p. 33)? TIPD evaluation 
must provide feedback and information 
that is useful for all stakeholders in their 
roles. The evaluation should be nested 
in the contextual realities in which the 
technology is integrated. It should take 
into account the dynamic nature of tech¬ 
nology development. To accommodate 
such demands, many researchers are 
advocating for collaborative evaluation 
approaches that are participant oriented. 
For such approaches, the evaluation 
methods are “iterative and character¬ 
ized by a process of experimentation, 
learning, and adaptation” (Patten, 2006, 
p. 33). Collaborative evaluation should 
build long-term relationships rather 
than short-term encounters. 

As stakeholders enlarge their 
evaluation lenses through collabora¬ 
tive evaluation, there will be an impact 
on professional development. Through 
collaboration, we can move beyond 
“inoculation” or “dump truck” models 
of professional development and focus 
our efforts on enhancing the success of 
ongoing long-term professional develop¬ 
ment strategies such as mentoring and 
situated modeling. In this way we will 
be providing support and feedback to 
teachers as they attain new knowledge, 
develop resiliency in the face of chal¬ 
lenges, and apply what they have learned 
on a long-term basis. 


Collaborative approaches to program 
evaluation emphasize emergent and par¬ 
ticipant-oriented designs. These designs 
are ideally suited for TIPD evaluations 
because they are more closely aligned 
with the unique characteristics of suc¬ 
cessful technology integration, including 
collaboration between multiple profes¬ 
sionals; the importance that contexts 
play in successful technology integra¬ 
tion; and the ever changing nature of 
technology equipment and resources. 
This paper will discuss three closely 
related methodological approaches 
(summarized in Table 1): developmental 
evaluation, responsive evaluation, and 
layered research. 

Developmental Evaluation 

Developmental program evaluations are 
designed to support program growth, 
continuous progress, and change. They 
potentially provide “rapid response to 
complex dynamic situations” (Patten, 

2006, p. 33) and are therefore “ideally 
suited for ongoing adaptation and rapid 
responsiveness” (Patton, 2006, p. 31) that 
leads to a reasoned program evolution. 

The goals of developmental evaluations 
are not generalization or exact replication 
of a program model, but to “discover and 
articulate principles of intervention and 
development” (Patten, 2006, p. 31) that 
program developers can apply to their 
respective programs. There are meth¬ 
odological and interactional differences 
with traditional models of evaluation. 

For example, in a traditional evaluation, 
goals and outcomes are routinely identi¬ 
fied prior to implementing the evaluation. 
In a developmental evaluation, rather 
than being preset, goals and outcomes 
emerge through the learning process. This 
emergent nature affects the relationships 
between program evaluators and imple¬ 
mented. In developmental evaluation, the 
evaluator is often an integral member of 
the program design team, not only vary¬ 
ing the evaluation methodology with the 
problem to be solved, but helping those 
involved in program design and imple¬ 
mentation to “think evaluatively” As such, 
developmental evaluation is tailored to the 
needs of all partners and is “designed to be 
congruent with and nurture developmen- 


94 I Journal of Digital Learning in Teacher Education I Volume 27 Number 3 

Copyright ©2011, ISTE (International Society for Technology in Education), 800,336,5191 (U.S. & Canada) or 541.302.3777 (Inf I), iste@iste.org, iste.org. All rights reserved. 



Technology Integration Professional Development Partnerships 


Table 2. Collaborative Evaluation Affects Professional Development and Partnership Relationships 


Evaluation Model 

How Model Influences Recursive 

Professional Development 

How Model Contributes to Sustained, 
Long-Term Relationships 

Developmental Evalu¬ 
ation 

Because evaluation is emergent and dialogic, 
new processes and topics for professional 
development can be implemented during the 
course of professional development rather 
than for future, new programs with new 
participants. 

Partners become co-designers. 

Ownership of and commitment to the 
professional development process can 
be sustained. Relationships can nurture 
new professional development opportu¬ 
nities within the partnership. 

Responsive Evaluation 

As evaluation focuses on the settings where 
learning occurs (i.e., both professional devel¬ 
opment and classroom contexts), evaluations 
can be accomplished across boundaries 
and focus on how teachers learn, regardless 
of the setting. There is greater potential for 
professional development to emphasize how 
participants learn. 

Partners become co-learners, serving 
as resources to one another both within 
and outside of the professional develop¬ 
ment context. 

Layered Research 

Participants and evaluators develop evalu¬ 
ation inquiries and data collection tools. 
Because participants’ inquiries inform the 
evaluation, professional development content 
can become authentic to participants’ work. 

Partners become co-researchers, 
focusing their relationship on developing 
new knowledge that can inform both 
classroom practice and the field as a 
whole. 


tal, emergent, innovative, and transforma¬ 
tive processes” among all those involved 
(Patton, 2006, p. 28). 

Dialogic processes facilitate the col¬ 
laborative, emergent nature of devel¬ 
opmental evaluations. Through dialog, 
evaluation participants make adjust¬ 
ments to evaluation design based on 
what is possible and what is desirable 
for the program being evaluated. This 
necessitates an “open-ended approach to 
data gathering, where the questions and 
concerns are emergent, and where trial 
and error is carefully mined for learn¬ 
ing” (Patton, 2006, p. 33). 

Responsive Evaluation 

Whereas developmental evalua¬ 
tions emphasize emergent design of 
evaluation to support the evolution 
of a program, responsive evaluation 
approaches (Stake, 1973) emphasize 
collaboration. Responsive evaluation 
reflects a “participatory democracy 
among stakeholders in evaluation set¬ 
tings” (Schubert, 2008, p. 403) and fo¬ 
cus on “the settings where learning oc¬ 
curs, teaching transactions, judgment 
data, holistic reporting, and giving 
assistance to educators” (Stake, 1973, 
p. 3). Like developmental evaluation, 
responsive evaluation does not begin 
with a preordinate approach. Rather, 

“it is evaluation based on what people 
do naturally to evaluate things: they 
observe and react” (Stake, 1975, p. 4). 
An educational evaluation is a respon¬ 
sive evaluation if: (a) it orients more 
directly to program activities than to 
program intents, (b) it responds to au¬ 
dience requirements for information, 
and (c) the different value-perspectives 
of the people at hand are referred to in 
reporting the success and failure of the 
program. In these three separate ways, 
an evaluation plan can be responsive. 
Due to the responsive nature, data 
collection methods are naturalistic and 
built on the perspectives of those in¬ 
volved. As such, ongoing observations, 
interviews, and document analysis are 
important components of responsive 
evaluations, as are the interpretations 
and meanings that collaborators make 
of the various forms of data collected. 


Layered Research 

Another evolution of collaborative 
evaluation, layered research (Burnaford, 
2006), explores how teacher participa¬ 
tion in research and evaluation informs 
the instructional decisions they make in 
their classrooms. As such, the evaluation 
and classroom contexts of a program are 
inextricably bound. In layered research, as 
teachers engage in program implementa¬ 
tion, they are “also engaged in document¬ 
ing and investigating their work with 
respect to student learning and their own 
professional development” (Burnaford, 
2006, p. 2). In traditional evaluation, the 
“agenda for evaluation is typically set by 
the stakeholders, which includes funders, 
board members, and consumers. The 
agenda shifts when participants in the 
implementation contribute as data gather¬ 
ers and analysts to that agenda” (Bur¬ 
naford, 2006, p. 3). This creates a milieu in 
which practitioners inform the knowledge 
base about what constitutes an effective 
classroom practice “while still requiring 
the evaluative procedures that provide 
them with direction for future planning” 
of professional development activities 
(Burnaford, 2006, p. 3). As such, an evalu¬ 
ation can be situated in classroom contexts 
and therefore has great potential to inform 
professional development programs that 
affect teacher and student learning. The 
core procedures for such an evaluation 


are documentation and action research 
(Burnaford, 2006). Through layered re¬ 
search, teachers frame their own research 
questions and engage in action research, 
documenting evidence related to their 
inquiries as a part of the action research 
process. Their action research inquiries 
are incorporated into the overall program 
evaluation. As such, teachers are “involved 
in continuous professional development” 
(Schubert, 2008, p. 403). 

These three collaborative approaches 
shift the processes used for program 
evaluation: from identifying inputs and 
measuring outcomes toward a contex¬ 
tually bound approach that is inclusive 
of the multiple perspectives of all stake¬ 
holders. These approaches are therefore 
ideally suited for TIPD. They expand 
the number of stakeholders involved in 
the evaluation and value dialogic pro¬ 
cesses, so they foster multiple defini¬ 
tions of success. The core processes for 
data collection, synthesis, and analysis 
are naturalistic and emergent and can 
therefore be derived from the unique 
dynamic contexts in which technology 
is integrated. Finally, the evaluation 
methodologies are designed to make 
teaching and learning visible in both 
professional development and class¬ 
room contexts. Therefore, the outcomes 
of professional development can be 
relevant to all involved stakeholders. 
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Each of these models of collabora¬ 
tive evaluation fosters unique oppor¬ 
tunities for the growth of professional 
development as well as benefits to the 
partnership relationships. Table 2 (p. 95) 
summarizes these aspects. 

Each of the collaborative evaluation 
models affirms qualities advocated by 
the National Staff Development Council 
(NSDC). Each model seeks information 
and feedback from multiple audiences 
and incorporates multiple sources of 
evidence so that findings can be relevant 
for all participants. Findings are made 
available during the course of profes¬ 
sional development so participants 
know the impact of their professional 
development experiences and act on 
those findings rather than waiting for 
the results of yearly testing. Finally, the 
NSDC maintains that audiences should 
have the “prerequisite knowledge and 
skills to interpret and use the infor¬ 
mation” (National Staff Development 
Council, n.d.). 

TIPD Evaluation Field Example 

Flow might these approaches look 
when grounded in a real evaluation? To 
explore this question, we draw from our 
own experiences developing, imple¬ 
menting, and evaluating a TIPD part¬ 
nership and share our lessons learned. 
Through a Preparing Tomorrow’s Teach¬ 
ers to Use Technology (PT3) grant pro¬ 
gram between a large urban university 
and a large urban public school system, 
we implemented integrated professional 
development and curriculum develop¬ 
ment with teachers in the public school 
system and teacher educators in the 
university setting. Our program con¬ 
sisted of a multi-week institute, ongoing 
professional development throughout 
the academic year, and a Web infrastruc¬ 
ture that supported teachers and teacher 
educators as they created Web-based 
classroom curriculum materials. As 
program implementers, our evaluation 
goals were to: 

• Provide feedback to program devel¬ 
opers and implementers that could 
be used to modify programs based on 
participants’ needs. 


• Assess change in participants’ knowl¬ 
edge, attitudes, behaviors of technol¬ 
ogy integration 

• Document impact on teacher practice 

Our evaluation model included 
pre/post surveys, observations, expert 
reviews, and artifact analysis. Pro¬ 
fessional development participants 
completed surveys prior to and follow¬ 
ing professional development summer 
institutes. Through their responses, we 
learned that teachers increased their 
knowledge of classroom-based technol¬ 
ogy integration practices, their attitudes 
toward the benefits of technology 
integration, and their (self-reported) 
behaviors toward integrating technol¬ 
ogy in their classrooms. Through this 
documentation, we learned that a given 
professional development activity was 
successful because it increased learners’ 
knowledge of the content, “heightened 
the value or importance they placed on 
integrating these new techniques, and 
increased the number of classroom- 
related behaviors related to these new 
practices” (Cunningham et al., p. 157). 
The results also provided feedback that 
was useful for adapting professional 
development sessions based on our 
program goals. 

Experts evaluated the materials that 
teams of teacher educators and class¬ 
room teachers created. This document 
analysis ensured that the modules cre¬ 
ated could be applicable and education¬ 
ally relevant in both teacher education 
classes and K-12 classrooms. In this 
way, document analysis served to secure 
connections between professional devel¬ 
opment and classroom contexts. 

Observations also helped situate the 
professional development sessions with¬ 
in classroom environments. To evaluate 
the quality of the lessons created during 
professional development, the research¬ 
ers developed and implemented an ob¬ 
servation protocol in participants’ class¬ 
rooms. Using the protocol, we evaluated 
lessons along a four-level continuum, 
from non-technology use to technology 
innovation. Results of these observations 
indicated that teachers were creating les¬ 
sons that employed more sophisticated 


uses of technology, a goal of the profes¬ 
sional development program. 

Through our evaluation methods, we 
learned that our professional devel¬ 
opment program increased teachers’ 
knowledge, attitudes, and behaviors to¬ 
ward technology integration. The quality 
of the lessons and materials used in both 
teacher education and K-12 classrooms 
also improved. These outcomes affected 
approximately 1,500 students, 300 teach¬ 
ers, and 35 teacher educators. 

Even though our professional devel¬ 
opment program was a collaborative 
that brought together stakeholders for 
professional development and curricu¬ 
lum design, the evaluation approaches 
discussed above were traditional and 
expert oriented. We did not evaluate 
the partnerships themselves, nor did we 
engage in collaborative evaluation pro¬ 
cesses. We never evaluated the collabo¬ 
ration between schools administrators, 
teachers, and teacher educators. We did 
not evaluate the relationships between 
each of these stakeholders and us as 
project implementers. Therefore, we do 
not know what nor how each of these 
groups learned from each other. Al¬ 
though the relationships between school 
and university partners were supportive, 
we often had difficulty integrating our 
various perspectives beyond the pro¬ 
gram activities. As such, it was challeng¬ 
ing to implement practices outside of the 
professional development context. 

We did not take a close look at the 
scaffolds we put in place for collabora¬ 
tion, such as the Web-based support ma¬ 
terials, technical support, and ongoing 
mentoring. Therefore, we were not able 
to learn how we could solidify longer- 
term relationships. 

Our evaluation enabled us to learn 
about the benefits of the various pieces 
of our program, but we lacked a holistic 
view because we drew upon limited 
perspectives to guide the evaluation. 

We did not include all the stakehold¬ 
ers as agents of the evaluation rather 
than passive respondents. As such, we 
missed the opportunity to guide our 
participants to become the sources of 
their own and each others’ professional 
development. Because of this missed 
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opportunity, four years later, we see that 
our relationships were not sustained, 
and without continued funding, long¬ 
term gains were not realized. 

How could the collaborative pro¬ 
cesses discussed above foster long-term 
success? We will examine some possibil¬ 
ities by synthesizing big ideas contained 
in the three approaches to collaborative 
evaluations discussed above. 

Big Idea #1: Collaborative evaluations 
build on the benefits of partnership; as 
such, collaborative evaluations can be 
guided by similar principles. 

The literature on partnerships indi¬ 
cates that those that are successful 
share certain features. For example, 
stakeholders should jointly con¬ 
ceive and agree upon program goals; 
teachers should be actively involved 
in the implementation process; each 
stakeholder must develop an apprecia¬ 
tion of the others’ contributions; and 
all should mutually understand and 
own the outcomes of a program (Cole 
& Knowles, 1993; Ravid & Handler, 
2001). These features can be fostered 
when evaluating a program as well. 
When sharing joint responsibility for 
program goals, participants can jointly 
define visions for what constitutes suc¬ 
cess and can drive the evaluation. As 
stakeholders become knowledgeable 
about the goals and processes of evalu¬ 
ation, program goals become more 
transparent to them and they can be¬ 
come invested in program outcomes. 

Collaboration creates the condi¬ 
tions for all stakeholders to be equally 
valued, and therefore all involved 
can achieve parity. If our program 
had incorporated the perspectives of 
our stakeholders early on, similar to 
developmental evaluations, we would 
have been able to build stronger rela¬ 
tionships with our participants, and 
our program could have developed in 
ways that reflected the needs of our 
participants. The outcomes of our 
program, including participants’ in¬ 
creased understanding and application 
of technological pedagogical content 
knowledge, could have been sustained 
over the long term. 


Big Idea #2: Collaborative evaluation can 
provide information useful for adjusting 
the program as it is being implemented. 

Through the process of implementation, 
ongoing interpretation of evaluation data 
can create a culture of problem posing 
between all of the stakeholders. In our 
PT3 evaluation, had we periodically cre¬ 
ated opportunities for our stakeholders 
to look over observational data, we might 
have been able to discuss patterns emerg¬ 
ing through the data and problemetize 
these patterns in ways that could have 
informed our professional development 
sessions. Our professional development 
activities might have become more mean¬ 
ingful and authentic to a larger popula¬ 
tion of our participants. In turn, this 
could have eased participants’ abilities to 
incorporate what they learned in the pro¬ 
fessional development context with their 
students, leading to higher-level technol¬ 
ogy integration in the classroom. 

Big Idea #3: Participants are involved in 
developing evaluation instruments and 
interpreting findings. 

Instruments used for evaluation data 
collection must be based on what 
constitutes success. When all stakehold¬ 
ers’ perspectives are brought to bear, 
the outcomes are relevant for them. 
When participants have input into the 
construction of data collection instru¬ 
ments, there is a greater potential for 
them to have clarity about the purposes 
for professional development. Because 
participants are a part of the data collec¬ 
tion process, a feedback loop is created 
that maximizes the likelihood that the 
evaluation can be responsive to their 
needs. Therefore, all stakeholders mutu¬ 
ally value and own outcomes. If we had 
framed the findings of our surveys and 
evaluations through the perspectives 
of all involved, stakeholders could have 
established learning relationships that 
would last despite changes in funding. 

Big Idea #4: Teachers frame their own 
inquiry questions related to the overall 
evaluation, and their data collection in¬ 
forms findings of the overall evaluation. 

Had we involved our participants in 
framing their own inquiries concerning 


their own professional growth, docu¬ 
menting their efforts, and analyzing 
student artifacts, as layered approaches 
advocate, there would have been greater 
potential for them to become sources 
of their own professional development 
as well as mentors to one another. By 
sharing observation tools with par¬ 
ticipants and gathering their feedback, 
we would have been able to develop a 
shared understanding of what success¬ 
ful technology integration means. We 
would be getting more feedback from 
the field, enabling us to garner multiple 
perspectives useful for ongoing program 
modifications as well as more holistic 
views of success. 

Conclusion 

Collaborative evaluations foster work¬ 
ing relationships and shared under¬ 
standings. They shift and expand the 
focus from the evaluation of outcomes 
only to the evaluation of processes and 
outcomes. By engaging stakeholders in 
both the processes and the outcomes of 
evaluation, professional development 
can be dynamic, responsive to the needs 
of a greater number of stakeholders, and 
sustainable over the long term. To be 
sure, funding must support collabora¬ 
tive evaluation. Traditionally funders 
support approximately 10% toward 
evaluation of a professional develop¬ 
ment program. This is only sufficient 
for inoculation forms of professional 
development and rarely leads to insights 
helpful for sustaining transformations 
in classrooms. When funders provide in¬ 
creased resources for broader and more 
inclusive evaluations, we will be able to 
create ongoing professional development 
partnerships that transform teaching 
and learning, better positioning students 
to engage in a technology-infused, flat¬ 
tened world. 
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